A Method for Evaluation of Learning Systems

The development of a method for evaluating the performance of learning systems is a prerequisite to their engineering applications. Such a method is also necessary for machine learning researchers for the objective evaluation and comparison 0/ the performance of individual experimental and commercial learning systems. This paper proposes such a method. based on the performance of a learning system in the encire multistage process of automated knowledge acquisition. The evaluation utilizes both the values of individual criteria selected/or a given evaluation and the character o/their learning curves. The description of the method includes its basic assumptions. the evaluation process, evaluation criteria and their classification. and an evaluation model based on the multi-attribute utility theory. Individual assumptions are justified. taking into consideration the results. produced using the AQ15 learning system on two col/ections of actual engineering examples from the areas of construction safety and structural design. Recommendationsfor further research and conclusions are also provided.